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ABSTRACT 

Three evaluation studies are used to illustrate the 
notion of an eclectic approach to adult education evaluation. The 
first study. Credit and Non-Credit offerings in a Federal 
Penitentiary, evaluated the learning climate in this educational 
setting using data collected from participating inmates 
(Questionnaire), non- participating inmates (interests inventory) ; 
educational counselors and teachers and program administrator 
(interviews) , and from Extension Division instructors (narratives) . 

In the second study. Consumer Notes (a weekly television program for 
homemakers) , a telephone survey conducted at the conclusion of the 
series and questionnaires completed by a panel of 30 homemakers 
weekly, at 13 weeks, and at the conclusion of the 2 6- week series were 
used to obtain evaluation data. Two learning modules, one on soil 
nutrients and soil testing and the other on communication, were 
evaluated using three evaluation nodes; reaction panel. Field Test A, 
and Field Test B. Three constraints which affected the choice of the 
evaluation strategies were (1) the kinds of questions which the 
programmer needed to answer, (2) the methods and sources of data 
collection, and (3) the limitations of resources in terms of time, 
money, and staff. Examples of evaluation models or elements are 
given, as follows: (1) Objectives-based evaluation, (2) Context 

evaluation, (3) Process evaluation, (4) Formative evaluation, and (5) 
Selection of criteria. (DB) 
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EducrAional accountability hat Tvcccme, within the last 
few years, a term common to professional educcitors and the 
public generally. While the more popular public debate tends 
to focus on the more formalized school systems, adult education 
in its many forms has and will come rtder scrutiny. Is the 
adult educator ready to respond? Do?s he have the expertise 
to be able to supply information to relevant decision-makers 
about these programs? Are the tools of evaluation available 
to him? This caper is concerned with demonstrating the more 
positive side to these questions by describing an eclectic 
approach to evaluation strategies. 

The body of literature on educational evaluation is 
.growing rapidly and is confusing to the uninitiated. In rece.it 
years there has been a large number of evaluation models 
developed and described in the literature. Nearly all of these 
have grown cut of evaluation projects and research centres 
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primarily concerned with the evaluation of the primary and 
secondary school systems. One might dismiss these approaches 
as being irrelevant if they have not been operationalized in 
programs similar to the ones we are evaluating. There is a 
need for adult educators to select elements of these models,, 
modify them, and use them in the contexts of adult education. 

A single moctel for evaluating adult education is not likely 
to emerge. What can and should develop, as is the case with 
formal educe tion, is a variety of approaches to evaluating 
adult education that make sense to the practicing adult educator. 

While the volume of material available continues to 
grow, the problems of evaluation don't necessarily ameliorate 
because of these references. Program decisions are made 
whether or rot reasonable evaluations are conducted. The task, 
then, is to become familiar with a variety of theoretical 
models of evaluation and make the appropriate selection and 
adaptation for the specific evaluation task. 

A number of the models of evaluation provide a framework 

12 3 

for evaluating total educational systems. * ' While these are 

* Stephen Klein, Gary Fenstermacher, Marvin C. Alkin. 

The Center's Changing Evaluation Model. Evaluation comment . 
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Los Angeles, 1971. 
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useful to the evaluator faced with the task of doing a 
comprehensive evaluation study, they can be bewildering to 
the practitioner wanting, for example, to determine whether 
certain instructional strategies are effective in a certain 
project. The adult educator also has more elusive sources of 
data than does the formal school evaluator. The learners are 
usually not sitting in a classroom, but may be at home listening 
to educatioral radio or television programs, attending a 
demonstration and dispersing immediately or some other equally 
temporary learning situation. 

Most adult educators are not content to run projects 
and wait until outcomes can be observed or measured before 
doing an evaluation. In fact, many outcomes may not be observable 
until the operation of the project becomes history. They want 
to build in formative evaluative strategies. This does not 
imply that adult educators are unconcerned with outcome 
evaluation but rather that they need to know whether adjustments 
in projects should be made before the project ends. Remedial 
learning situations are not as easy to structure in adult 
education programs as they are in the formal classroom. 

Many adult education projects are developmental in nature 
and consequently have changing objectives or objectives stated at 
non-operational levels. The evaluator concerned only with 
measuring goal attainment will find this situation an 
incomprehensible one. 
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The evaluator of adult education programs will find 
the selection of criteria a major problem. Decision-makers 
using the evaluative data tend to be non-committal about the 
criteria they will use in making judgments. Standards are 
difficult to ascertain. 

Stufflebeam et al suggest a number of symptoms of 
evaluation's illness. Not the least mentioned in the first 
chapter of their book is the lack of certain crucial elements 
needed if evaluation is to make significant forward strides. 

Not only is the lack of identification of these 
elements a drawback but so is the extent to which those 
identified have been operationalized. It is through application 
that refinement and modification of theoretical models can 
occur . 



P.K.D. Study Committee on Evaluation, op . cit . 
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This section of the paper describes three evaluation 
studies. Each project being evaluated requires different 
kinds of questions to be answered. The evaluation strategies 
were designed to provide data that would assist the project 
> directors in making decisions that were imminent. 

1 . Credit and Non-Credit Offerings in a Federal Penitentiary 
The Extension Division of the University of Saskatchewan 
contracted with the Canadian Penitentiary Services to provide 
classes in a federal penitentiary. This was the first occasion 
for the Division to be formally invoLved in the inmate 
educational enterprise. Also, this was the first occasion to 
have formally organized classes offered to inmates and taught by 
instructors other than educators in the employ of the penitentiary. 
The major concern of the project supervisor was to evaluate 
the learning climate of this educational setting. It was 
decided that relevant information could be obtained from the 
following groups: participating inmates, non-participating 

inmates, educational counsellors and teachers of courses within 
the penitentiary system, instructors of the university courses, 
and the director of education services for this penitentiary. 

For each course held all those who attended were invited 
to complete a questionnaire reacting to the method of instruction, 
their perceptions of the instructor, how they felt in these 
courses, and the level at which the material was taught. The 
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questionnaire was confidential and turned in directly to 
the Extension Division staff. In addition, a few inmates 
communicated directly, either verbally or by letter, to the 
Extension Division about these courses. 

Interviews were conducted with educational counsellors 
and teachers in the penitentiary to determine their perception 
of the effect these classes had on participating inmates. The 
administrator was interviewed to determine the constraints which 
the Federal :?enitentiary System might have on such educational 
endeavour s . 

The other inmates not enrolled in these classes were 
encouraged to complete an inventory of their interests and 
motives regarding educational courses. 

The instructors hired by the Extension Division provided 
a narrative about their perceptions of what was going on in the 
classroom. 



2. Consumer Notes - a weekly television program for 
homemaker s . 

In the spring of 1971 a telephone survey was 
conducted at the conclusion of a winter's program of Consumer 
Notes. The primary purpose was to determine the extent to which 
the program was. viewed by homemakers. The study yielded 
sufficient information which was used to make revisions in the 
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program. But one of the concerns was to be able to get viewer 
reaction while the series was in progress so that this feedback 
could be used in making program improvements. An evaluation 
design was implemented which made use of a panel of 30 homemakers 
throughout the seven viewing areas of the province. These 
women were selected by a contact in each community with an 
attempt made to have background variables taken into consideration. 
Each panelist was provided with token remuneration. Following 
each weekly program each panel member completed a questionnaire 
and mailed it to the Extension Division. The questionnaire 
attempted to get information about the panel member ' s reactions 
to the quality of the production, the subject matter of the 
program, and the personalities involved in the program. Also 
included was a section asking about recall from the previous 
week's program, if viewed, and whether any use had been made of 
the information presented in the program. At the mid-way point 
of the 26-week series and at the conclusion, each panel member 
completed a questionnaire asking for more general reaction to 
the series and having them priorize programs in terms of 
explicit criteria. 

The c.bove procedure was designed primarily to provide 
for formative evaluation of the series. In addition, a telephone 
survey was conducted following the conclusion of the series so 
that comparisons could be made with the results of the survey 
conducted the year previously. 

ERIC 
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3 . T he Production of a Learning Module 

The Extension Division produces learning modules 
in a variety of subject matter areas. The presentation of 
the material may be on slide-tape, on video tape, on 16mm film, 
overhead transparencies, or combinations of these. During 
the past year two of these programs were evaluated. One was 
on soil nutrients and soil testing presented on videotape. 

The other was on communication and was presented through the 
medium of three slide projectors synchronized with cassette 
tape. 

The production of a learning module from conception to 
the final product requires some mechanisms which provide for 
adaptations in the module during production. The formative 
evaluation strategies need to be recognized by all those 
involved in producing the module (the content specialist, 
the media specialist, the adult educator). It seemed 
unreasonable to expect those producing the learning module to 
revise continuously without sufficient data to make the 
decision for revision. The following model provides the basis 
for evaluating learning modules. 
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Figure 1. A representation of the evaluation strategy used' 
for learning modules. 



There are three evaluation nodes in this model: 

1. Reaction panel - When a module reaches a stage where 
both content and process have been put together 
(normally not in a form that has reached any final 
decisions) a reaction panel is selected to be exposed 
to the module. The panelists are normally other 
members of faculty and should have representation 
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from both content and educational process. 

Suggestions for revision are recorded. Usually the 
people producing the package are present at the 
showing and can receive the information directly. 

2. Field Test A - This is conducted under selected 
circumstances. The participants recognize they 
are involved in a "first run. " They should be 
reasonably representative of the type of audience 
for which the module is produced. Data are collected 
about the reactions to the learning process they 

are experiencing as well as the extent to which the 
learning objectives are reached. These are used 
as a basis for further revision to the module. 

3. Field Test B - The module is used under field 
conditions. Pre and post tests are administered 
to determine change in behavior. Items in the 
testing instrument for which expected outcomes 
are not evident are noted so that changes in that 
section of the module can be made. 
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The Selection and Application of Evaluation Strategies 

The three evaluation studies described in this paper 
will be used to illustrate the notion of an eclectic approach 
to evaluation. 

The evaluator, in making decisions about the design 
of the study, has several constraints which, in a sense, provide 
the limitations in choosing an appropriate design. Three of 
these which were in effect in choosing strategies for the 
three studies reported here are as follows: 

1. he kinds of questions which the programmer 
needed to answer, in the penitentiary project, 
for example, the concern was centred around the 
extent to which the environment in which the 
inmates were studying interfered with a good adult 
learning climate. On the other hand, the primary 
concern in the evaluation of learning modules was 

to what extent the learning package was reaching its 
stated objectives. 

2. The methods and sources of data collection. 

* 

One might feel that having framed the appropriate 
questions for the study one should select the obvious 
method of data collection and sources of data, but 
this is not always possible, in the penitentiary 
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project, it was a deliberate decision not to collect 
data about the non-credit classes through tests or 
exams. In the television project, it was not possible 
to interview people on a weekly basis although that 
method of data collection may have yielded more 
qualitative data for program revision purposes. 

3. The limitations of resources in terms of time, 

money, and staff. Most adult educators find them- 
selves wanting to evaluate projects with a severe 
time constraint. We had to know whether to continue 
the offering of educational programs in the 
penitentiary the following year almost at the same 
time as the current courses concluded. Typical of 
most practitioners, the funds available to 
evaluate projects in the Extension Division are 
often not included in budgets. In the case of the 
television project and the learning modules, no 
evaluation budget was specified and therefore operated 
under ad hoc arrangements for financing. The 
penitentiary project did have a budgeted item for 
evaluation. The expertise of staff offers further 
limitations. One tends to select strategies which 
are familiar and can be operationalized without 
hiring consultants. 
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These three constraints, then, are used for screening 
models or elements of models in building an appropriate 
evaluation design. 

Some Examples of Selection of Models or Elements 

1. Obiectives-based evaluatio n 

The Tylerian congruence model is used in the final 
stages of evaluation of learning modules. It is at these 
stages where objectives are explicated in specific behavioral 
terms and data are collected from learners to determine to 
what degree these objectives are reached. 

2. Context evaluation 

The context dimension of the CIPP model^ was the 
major source for designing the evalueition for the penitentiary 
project. Perhaps, more specifically, the congruence element 
of context evaluation (comparing actual and intended system 
performance) was used, although the contingency mode (searching 
for opportunities and pressures outside the system to promote 
improvement within it) helped us in determining additional 
sources of information. 



^P.K.D. Study Committee on Evaluation, op. cit . pp. 218-220. 
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3 . Process evaluation 

The monitoring of the Consumer Television Series 

and the ongoing classes in the penitentiary fall into the 

process evaluation mode. The two models providing assistance 

in designing the strategy for these concerns were the process 

element of the CIPP model^- and the transactions element of the 

2 

Stake model. 

4. Formative evaluation 

3 

Formative evaluation as outlined by Scriven was 
of primary consideration in all three evaluation studies. In 
the penitentiary project we were looking for data to use 
immediately in making judgments about course content, in-class 
strategies, and administrative arrangements. The weekly mail-in 
reaction sheets from the panelists in the consumer television 
series were used in making decisions about upcoming programs. 

The need for reactive data for use in developmental stages of 
replicated forms of learning experiences is obvious. This is 
the reason for having at least three evaluation nodes in the 
evaluation design for learning modules. 

^■p.K.D. Study Committee on Evaluation, op. cit . pp. 229-232. 
^Stake, op. cit . 

3 

Michael Scriven. M The Methodology of Evaluation. " 

AERA Monograph Series on Curriculum Evaluation No. 1. 

Rand-McNally, Chicago, 1967. 
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5. . Selection of criteria 

The judgment matrix of Stake's model”*' provides a 
basis for collecting data about standards and criteria, in 
the penitentiary project the interviews with the Director of 
Education, the school counsellors, and the report forms from 
instructors all yielded data for generating criteria. 

The first reaction panel in the learning module 
evaluation project is generally a rich source of standards 
and criteria for that particular module. 

The foregoing examples illustrate the versatility of 
various models of evaluation if one chooses to be eclectic. 
The selection of various elements from models is not always 
a conscious decision. However, it is reasonable to suggest 
that one is more likely to develop more rational approaches 
to evaluation design when a variety of alternatives are 
considered than when one forces all evaluation projects into 



a favored mode of evaluating. 
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